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The purpose of this article is to define and explain a trend that has 
caused a great deal of confusion among HR researchers, practitioners, 
and consumers of HR-related services: competency modeling. The 
Job Analysis and Competency Modeling Task Force, a work group 
jointly sponsored by the Professional Practice Committee and the Sci- 
entific Affairs Committee of the Society For Industrial and Organiza- 
tional Psychology, has recently concluded a 2-year investigation into 
the antecedents of competency modeling and an examination of the 
current range of practice. Competency modeling is compared and con- 
trasted to job analysis using a conceptual framework (reflected in a 
10-dimension Level of Rigor Scale) that practitioners and researchers 
may use to guide future work efforts, and which could be used as a basis 
for developing standards for practice. The strengths and weaknesses of 
both competency modeling and job analysis are identified and, where 
appropriate, recommendations are made for leveraging strengths in 
one camp to shore-up weaknesses in the other. 
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The business environment today is characterized by incredible com- 
petition and change (D'Aveni, 1994; Hamel & Prahalad, 1994). In re- 
sponse, organizations are flattening, relying on self-managed teams with 
greater frequency, becoming highly matrixed, and otherwise reconfigur- 
ing the structure of work (Ashkenas, Ulrich, Jick, & Kerr, 1995; Howard, 
1995; Keidel, 1994). Accompanying these changes has been a growing 
concern that traditional job analysis procedures may be unable to con- 
tinue to play a central role in the new human resource management en- 
vironment (Barnes-Nelson, 1996; Olian & Rynes, 1991; Sanchez, 1994). 
It is with this backdrop that the practice of competency modeling has ex- 
ploded onto the field of human resources over the past several years, To- 
day, surveys of competency-based practice indicate between 75% (Cook 
& Bernthal, 1998, based on a survey of 292 organizations) and 80% 
(American Compensation Association, 1996, based on a survey of 426 
organizations) of responding companies have some competency-driven 
applications currently in place. 

Given the turbulent practice environment, and the magnitude and 
pace of the growth of competency modeling, it is not surprising that 
practitioners and consumers of human resource services alike are look- 
ing for some meaningful reference points to guide their work, lb aid 
in this effort, the Society for Industrial and Organizational Psychol- 
ogy (SIOP) commissioned a task force in September 1997 to investi- 
gate and review the practice of competency modeling. The members of 
the SlOP-sponsored Job Analysis and Competency Modeling Tksk Force 
( JACMTF) 1 have conducted an extensive literature search, interviewed 
37 subject matter experts (SMEs) from varying backgrounds in the de- 
velopment and use of competency models, and have drawn on a rich base 
of personal experiences to shed light on questions such as: 

• What is a competency? 

• What is the difference between competency modeling and job anal- 
ysis? 

• Why are competencies so appealing to consumers in business and 
industry? 

• What is the future of competency modeling? 

The purpose of this article is to communicate the descriptive findings 
of the task force, and to offer suggestions for guiding research and im- 
proving practice in both competency modeling and job analysis. These 
suggestions are framed around a conceptualization of evaluative criteria 
that could eventually serve as a basis for standards for practice. 



1 Adjunct members of the task force who provided significant input included Tim Athey, 
Thomas Braun, Steve Doerflein, Richard Lonetto, John Morrison, Earl Nason, Yvette 
Tazeau, and Robert Tett. 
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Literature Searches 

Both computer-based and manual searches of published research 
and reviews focusing on competencies were conducted. The computer 
databases of the American Psychological Association, UMI Proquest Di- 
rect, Harvard Business Review, and the American Management Associ- 
ation were used to identify articles, dissertations, and book chapters that 
included analyses or discussions of the concept of competencies. The 
manual review included examining the proceedings from conferences 
devoted to competencies or competency modeling, government techni- 
cal reports, conference presentations, books, consulting publications and 
materials, and unpublished research and reviews. 

SME Interviews 

The JACMTF identified six potentially distinct perspectives that 
could be brought to bear on the issues surrounding competencies and, 
for each group, interviewed five to nine SMEs. 
L Human resource consultants or professionals who either develop 
competency model systems, or who are knowledgeable consumers, 
yet have no I-O background or link to SIOP (n - 9). 

2. Visible thought leaders in the area of competency modeling from 
either a pro or con perspective (n = 6). 

3. Former presidents of SIOP (n = 5). 

4. Industrial and organizational psychology (I-O) types who represent 
a traditional job analysis perspective (n = 5). 

5. I-O psychologists who have delivered both traditional job analysis 
and competency modeling projects (n - 6). 

6. International consultants who operate outside the United States and 
work with non-U. S. -based companies (n = 6). 

These six perspectives formed the basis for a 6-segment sampling 
plan. In all, 37 interviews were completed using a structured interview 
format so the SME responses could be meaningfully compared and con- 
trasted. The interviews were conducted by task force members, via tele- 
phone, over a 1 -month period* Of these 37 SMEs, there were some cases 
when an individual might have represented or had experience with more 
than one of the six sampling categories. However, the decision was made 
to include SME input in one, and only one, category. The questions 
posed to SMEs included: 

• How do you define a competency? 

• What types of human resource applications can competencies and 
competency modeling support? 
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• What types of human resource applications should competencies and 
competency modeling not be used to support? 

• How is competency modeling different from job analysis? 

• Should competency models be validated? 

• Why competencies? What is the attraction/value beyond other ways 
individual differences might be described and characterized? 

• Where do you see the future of the competency modeling field 
headed? 

The complete interview protocol, including follow-up question probes, 
is reproduced in the Appendix, 

What Is A Competency? 

To begin with, the word "competencies" today is a term that has no 
meaning apart from the particular definition with whom one is speaking 
(Zemke, 1982)* Some examples of efforts to define the term from SMEs 
representing each of the groups in the sampling plan include: 

• "The knowledge, skills, and attributes that differentiate high per- 
formers from average performers." 

• "Competencies are not fundamentally different from traditionally 
defined KSAOs (i.e., knowledge, skills, abilities, and other charac- 
teristics)." 

• It is a construct that helps "define level of skill and knowledge." 

• "Observable, behavioral capabilities that are important for perform- 
ing key responsibilities of a role or job." 

• "Mishmash of knowledge, skills, and abilities and job performance 
requirements." 

• "I can't" 

Some of the more frequently cited definitions from the literature 
include: 

• A mixture of knowledge, skills, abilities, motivation, beliefs, val- 
ues, and interests (Fleishman, Wetrogen, Uhlman, & Marshall-Mies, 
1995). 

• A knowledge, skill, ability, or characteristic associated with high per- 
_ fcmiwn^^a^job (Mirabile, 1997). 

• ^^ ^abj^ tig^bf motives, traits, self -concepts, attitudes or values, 
content knowledge or cognitive behavior skills; any individual char- 
acteristic that can be reliably measured or counted and that can be 
shown to differentiate superior from average performers (Spencer, 

^_ McClelland, & Spencer, 1994). 1 

• A written description of measurable work habits and persona] skills 
used to achieve work objectives (Green, 1999). 
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Clearly, there is a wide range of def initions, even among a fairly ho- 
mogeneous expert population, underscoring the difficulty of pinpointing 
a standard definition of the term. This lack of consensus shouldn't be 
too surprising, given the multiple domains in which the terms "compe- 
tent" or "competency" are prevalent. For example, the extensive use of 
these terms just in the early psychological literature is evident from the 
large number of hits (over 1,300) returned from a search for "compe- 
tency" in the pre 1966 Psychlnfo databases. In part, these words have 
their origins in law and, later, in clinical psychology, where the term 
evolved to define legal standards of mental capacity and awareness, the 
ability to care for oneself or others, and/or the ability to function in mul- 
tiple activities of "daily living." Subsequently, the term "competency" 
was embraced in the vocational counseling profession to define broad 
areas of knowledge, skills, and abilities linked to specific occupations. 
The word also has an extensive history in the field of education with an 
emphasis on broader traditional "knowledge" areas (e.g., mathematics, 
English). Early industrial psychologists also used the term "competent" 
to describe successful individuals in specific professions. In all of the 
above contexts — legal, clinical psychology, vocational, educational, and 
industrial psychology — the term "competence" defines "successful" per- 
formance of a certain task or activity, or "adequate" knowledge of a cer- 
tain domain of knowledge or skill. 

The history of competency modeling practice has followed a similarly 
circuitous path. Multiple parallel domains contribute to the evolution of 
the practice of competency modeling, including: 

• Individual differences and educational psychology. 

• Leadership research and the history of assessment centers. 

• Job analysis research. 

• The concept of multiple intelligences. 

• Prahalad and Hamel's (1990) concept of "core competency." 
Each of these foundational building blocks are briefly described below. 

Individual differences and educational psychology. The study and 
examination of individual differences is as old as modern civilization. 
Aiken (1988) cites attempts from the Bible and ancient history to iden- 
tify and label differences in human behavior and to use those differences 
for a specific purpose. In the history and systems of psychology, two 
major approaches characterize the conceptualization of human perfor- 
mance. McLagan (1996) describes these approaches as the differential 
psychology approach and the educational/behavioral approach. The for- 
mer focuses on capabilities or characteristics that are relatively endur- 
ing and manifested early* In the late 19th and early 20th century, Galton 
and Cattell pioneered the development of objective techniques to mea- 
sure human abilities and characteristics. These early efforts focused on 
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a means to measure intellect and, in particular, focused on identifying 
specific sensory and psychomotor abilities underlying intellectual func- 
tioning. The science of individual differences through the 1950s and be- 
yond quickly expanded to multiple and sometimes overlapping research 
domains: physical, intellectual, information processing, motivation, per- 
sonality, values, and more recently, emotional characteristics (see Guil- 
ford, 1956; Fleishman & Quaintance, 1984; Rokeach, 1973). Each do- 
main was studied using a wide variety of methodologies and techniques, 
but all were based (in whole or in part) on inferences from behavioral 
manifestations. These manifestations were in turn grouped and labeled 
through judgment or quantitative methodology, or some combination of 
the two. 

Although the differential approach focuses primarily on innate abil- 
ities, the primary emphasis from the educational psychology perspective 
is on performance outcomes and shaping behaviors so that people can be 
successful. Researchers in this camp have been concerned with creating 
educational strategies to develop successful performance. Bloom's work 
(1956, 1967; Krathwohl, Bloom, & Masia, 1964) to create a taxonomy of 
educational objectives, and Gagn6's (1975) efforts to use taxonomies for 
clarifying objectives for individual development are examples of work in 
this area. In most cases, the goal is to operationally define the taxonomic 
categories with illustrative, observable behaviors, which is the same tack 
taken in most competency modeling approaches. 

Of course, the field of industrial and organizational psychology re- 
lies heavily on an assumption inherent in both of the approaches de- 
scribed above — namely, that an individual's standing on many of the 
above-mentioned individual difference dimensions and/or knowledge, 
skills and abilities (learned, acquired, or enhanced) have the potential 
to predict job performance or success. 

Leadership research and assessment centers. The identification and 
assessment of characteristics underlying successful management perfor- 
mance and leadership behavior has a rich and varied history (Bass, 1990; 
Bentz, 1963; Laurent, 1961; 1962; 1968; Spreitzer, McCall, & Mahon- 
ney, 1997; Tkylor, 1960). Within this context, the assessment center ap- 
proach is one of the many procedures that have been developed to sat- 
isfy the interests and requirements of business and industry for selecting 
managers and leaders (Bray, 1982). According to Thornton and Byham 
(1982), military assessment programs in World War I and World War II 
(OSS, 1948), early personality research, and leadership/supervision job 
analysis research all served as the basis for the development of the man- 
agement assessment center, which was originated in the AT&T manage- 
ment progress study. The original dimensions were selected based on a 
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review of the management literature and the judgments of AT&T per- 
sonnel staff. 

An interesting observation that can be made upon reviewing the as- 
sessment center literature is the almost controlling influence the original 
set of dimensions derived from the AT&T research had on the assess- 
ment center field; resulting in a curious homogeneity across organiza- 
tions in the dimension-level taxonomies used to represent job content 
in different assessment centers (Schippmann, Hughes, & Prien, 1987). 
The implicit assumption seemed to be that there was a great deal of sim- 
ilarity in management functions across organizations and levels of man- 
agement. Further, an additional consistency in the assessment center 
programs, which in part is due to the fact that the taxonomies were ho- 
mogeneous in the first place, is that the dimension categories are very 
broad and generic. These dim ensions seemed to s erve as labels for clus- 
ter sof "attributes " ^^Ealracteristire^ ig *^ crit ~ 

jcjljfor^okiuc^ what ar e gomjen tionally called "corF 

structs " (e.g., drive, planning, creativity, flexibihty). In many waysTffie 
dimensional structure of assessment centers, and the resulting opera- 
tional definitions of the broad, generic individual difference dimensions 
using behavioral statements, was a portent of things to come in the realm 
of competency modeling. 

Consider Finkle's (1976) work to provide a comparison of the as- 
sessment dimensions from several early assessment center programs: 
Bray and Grant (1966), Hinrichs (1969), Thompson (1970), DiCostanzo 
and Andretta (1970), and McConnell (1969). An assessment dimen- 
sion like Drive (or Energy), which is common to these five distinct as- 
sessment center programs, looks very similar to the competency dimen- 
sions found in the generic models of well known providers of com- 
petency modeling services like Lominger (Drive for Results; see Lo- 
mardo & Eichinger, 1998), Hay-McBer (Persistence; see Spencer & 
Spencer, 1993), and Personnel Decisions International (Drive for Re- 
sults; see Davis, Hellervik, Skube, Gebelein, & Sheard, 1996). Com- 
mon assessment center dimensions like Planning (Bray & Grant, 1996; 
Hinrichs, 1969; DiCostanzo & Andretta, 1970; McConnell, 1969), cre- 
ativity/innovation (Bray & Grant, 1966; Thompson, 1970; McConnell, 
1969), flexibility (Bray & Grant, 1966; DiCostanzo & Andretta, 1970; 
McConnell, 1969), and many others, also have parallel dimensions in the 
competency libraries of Lominger (e.g., Planning, Creativity, and Deal- 
ing with Paradox), Hay-McBer (e.g., Systematic Planning, Innovation, 
and Flexibility), and Personnel Decisions International (e.g., Establish 
Plans, Innovate, and Demonstrate Adaptability). Though not the major 
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thrust of their paper, additional information about similarity in compe- 
tency structures across the generic models used by several major con- 
sulting firms may be found in a forthcoming article by Tfett, Guterman, 
Bleier, and Murphy {in press). 

Job analysis* One way to define job analysis is as a set of procedures 
designed to identify and describe those aspects of performance that dif- 
ferentiate most sharply between better and poorer workers (Anastasi & 
Urbina, 1997), TOs_definition is virt ually the same language frequently 
used^^fine^craipetency modefingTTfhis fociiTcmthe cnficaljob re- 
quirements that _di8eiCTfia^^E5H"Sx)m low performers ledTo^ie^e- 
velopment of the worker-oriented job element method of job analysis 
(Primoff, 1975; Primoff & Eyde, 1988) that was employed by Primoff 
and his associates at the U.S. Office of Personnel Management since the 
mid-1950s. Interestingly enough, one of the specific approaches spurring 
much of the early interest in competency modeling is the Hay-McBer 
approach (Spencer et aL, 1994), which is essentially an adaptation and 
extension of the job element method of job analysis. 

It should also be noted that, although the task force identified compe- 
tency modeling work for virtually all major job groups, much of the initial 
work seemed to have involved manageme ntpopulatio ns. This perspec- 
tive is supported by recent surveys bt competency modeling practices 
(American Compensation Association, 1996; Cook & Bernthal, 1998). 
Given this context, job analytic efforts to define the job performance 
domain of management work become relevant 

The difficulties in precisely defining the job of manager (i.e., real 
differences in management/supervisor level, long-term cycle of activi- 
ties, activities rarely repeated or duplicated, organizational and envi- 
ronmental demands change job demands) created a demand for inno- 
vative techniques in job analysis. For example, in the 1940s and 1950s, 
Fleishman (1953) and Flanagan (1954) systematically analyzed supervi- 
sor job behavior and identified broad factors or dimensions of perfor- 
mance. Fleishman, and others in the research group at The Ohio State 
University (Halpin & Winer, 1957), utilized 1,800 statements generated 
by expert judges and quantitatively reduced these items to two factors: 
consideration and initiating structure. These two behavioral dispositions 
served as the basis for the predominant view of supervisory behavior in 
the management literature as well as two well-known research question- 
naires: the Leader Behavior Description Questionnaire and the Leader- 
ship Opinion Questionaire. At about the same time, Flanagan pioneered 
the classic "critical incident'* technique through extensive studies of crit- 
ical incidents of effective and ineffective performance in Air Force offi- 
cers. Flanagan identified six broad categories of performance behavior 
through analysis of critical incidents provided by 3,000 officers. 
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In the years since, multiple researchers followed with increasingly 
more sophisticated and complex taxonomies of managerial performance 
and behavior. Hemphill (1960), Katzeil, Barrett, Vann, and Hogan 
(1968), Tornow and Pinto (1976), Yukl and Lepsinger (1992), Borman 
and Brush (1993), and numerous others have all made contributions in 
the past 45 years. Although each contribution is unique, all utilized sys- 
tematic questionnaires, interviews, observations, and large samples in 
order to subsequently reduce (usually through quantitative procedures) 
a large number of descriptor statements to a small number of job perfor- 
mance dimensions or categories. Further, although there is some unique 
variability in the dimensionality across these studies, what is more strik- 
ing is the substantial overlap in terms of job performance categories. 
In other words, although descriptor items or behaviors may be unique, 
there is a good deal of commonality in the broad dimension labels and 
organizing structure of the "results/' 

Along these lines, it should be noted that Boyatzis (1982) extended 
the work of McClelland (1973; see below) to managerial jobs. He reports 
a study of 2,000 persons in management jobs from 12 organizations and 
proposes 2 1 "characteristics" jn his management competency model. 
Boyatzis used behavioral event interviewin g (a variation on the critical 
incident technique), a picture-storyTechnique and a learning style in- 
ventory. Although his methodology and conclusions are controversial 
and have been criticized (Barrett & Depinet, 1991), his results are not 
dissimilar with the taxonomic conclusions of many of the job analytic re- 
searchers working with management populations. 

Furthermore, Tett, Guterman, Bleier, and Murphy (in press) have 
recently completed an effort to develop a taxonomy of managerial "com- 
petencies" derived from earlier models. Using a series of sorting tasks, 
these authors found that content experts could reliably sort behavioral 
statements into 47 dimensions; thereby establishing the meaningfulness 
of the categories and satisfying a critical criterion of taxonomic evalua- 
tion (Fleishman & Quaintance, 1984). 

The concept of multiple intelligencies. In 1973, David McClelland 
wrote an influential article entitled ''Testing for Competence Rather 
than for Intelligence/' His assertions revolved around the theme that tra- 
ditional intelligence and aptitude tests were not related to important life 
outcomes. He proposed "competency testing" or "criterion sampling' 7 as 
a viable replacement. Conn^etendeyaccor ding to McClellend, appea r 
to be know l^dggi^glriJlSr-abilities, resembling* 
or related to the job or job p e rformance or so m e other important life 
outcome. 

A program of research conducted by Gardner also stimulated in- 
terest in a "competency-like" approach. Gardner used neurological 



712 



PERSONNEL PSYCHOLOGY 



research with gifted and brain-damaged populations to propose a the- 
ory of "multiple intelligences" (e.g., Gardner, 1983; Shapiro, Grossman, 
& Gardner, 1981), in which intelligence is a dynamic concept involv- 
ing multiple competencies and opportunities by society to develop these 
competencies. Like McClelland, Gardner suggested that a wide variety 
of human cognitive competencies exist and can be examined more ef- 
fectively if "culturally valued performances are assessed in naturalistic 
settings" (Hatch & Gardner, 1986). 

Prahalad and HameL The immediate predecessor to, and likely the 
driver of, the current surge of competency modeling practice involves the 
field of business strategy and a concept known as "core competencies." 
As conceived by Prahalad and Hamel (1990), core competencies are not 
individual-level attributes as much as design components of an organi- 
zation's competitive strategy. Their work mirrors, at an organizational 
level, the identification of fundamental (and unique) KSAOs that drive 
an organization's ability to rapidly change and innovate in response to 
new and changing markets. Actually their use of the term "competency" 
is, to some extent, arbitrary and refers primarily to "the collective learn- 
ing in the organization." 

Despite being a distinct concept, the huge popularity of "core compe- 
tencies" in American business (Prahalad & HamePs 1990 Harvard Busi- 
ness Review article is the most reprinted paper in the journal's history) 
likely galvanized much of the interest in individual-level competencies 
by highlighting the importance of "people-embodied skills" necessary 
to building core competence, Lawler (1994) expands on these ideas 
and calls for a nontraditional approach to KSAO identification based on 
"organizational" analysis and effectiveness rather than job analysis and 
effectiveness. Lawler does not necessarily call for techniques uniquely 
different from job analysis. Rather, he calls for an assessment of knowl- 
edge, skills, and abilities that would allow an employee to successfully 
perform any number of tasks and jobs. 

Be that as it may, the concept of "core competency" has had an enor- 
mous amount of influence on the world of human resources. The nat- 
ural implications of Prahalad and Hamel's (1990) ideas for human re- 
source management created a natural market for a methodology to iden- 
tify these "human skills that embody core competencies." This demand, 
combined with the extensive historical use of the term in multiple, highly 
visible domains, seems to have brought competency modeling into com- 
mon, lay use as a generic "fusion" of concepts into a practice for identi- 
fying job requirements that is not altogether different from job analysis. 
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What Is The Difference Between Competency Modeling And Job Analysis? 

The SIOP task force members posed the question above to the SME 
group, and the results were intriguing. Although several subject mat- 
ter experts believed that competency modeling and job analysis are the 
same, by far the greater number stated that the processes are differ- 
ent. For example, th e majority of experts bel ieve that job analysis is 
much m o r^orkramg^^fogus ed and compet^cytnodgKng is works r- 
f^j^JTIaother words, job analysi s may be thought of as primarily look- 
ing at ' fwhav is accompiisRed , and competen cy modeling focuses more 
on / ^iow^ objectives are met or how work is accomplished. There are a 
number of other distinctions as well. The question then becomes how 
best to organize and describe the similarities and differences between 
the two approaches. 

One vantage point from which to make the comparison is with re- 
spect to the variables that may affect the results of the job analysis or 
competency modeling effort By that we mean the inferential leap from 
the product of the research, and the effort to utilize the product to guide 
decisions about the development of a human resources application, is 
either widened or reduced based on the presence of these variables. As 
with job analysis, the results of a typical competency modeling effort are 
seldom viewed as the end product. So, for both competency modeling 
and job analysis, there is usually some back-end application or set of 
applications that will be modified or constructed based on these results. 
Thus, errors in this front-end research cascade down into some unknown 
amount of error that gets built in to the resulting application. Given this 
fact, there has been a widespread implied assumption among human re- 
source professionals that the front-end modeling or analysis activities 
must meet some minimal standards of acceptability so that inferences 
from the product of the method can be confidently drawn. One way to 
compare and contrast the "typical" competency modeling effort from 
the "typical" job analysis effort lies in the level of effort devoted to man- 
aging these variables (i.e., the level of rigor of the research methodol- 
ogy). From this perspective, these variables might essentially serve as 
evaluative criteria. With this goal in mind, the members of the JACMTF 
identified and defined 10 such variables. 

The fir st variable of this type concerns the meftod^oUDxestigation. 
In other words, what is the thinking behind the selection of the method 
(e.g., interviews, focus groups, observation, questionnaires, electronic 
monitoring, or work diaries) for collecting information? For example, a 
given method may have the potential for yielding an accurate description 
of a largely physical job but be incapable of investigating all aspects of a 
primarily mental job. 
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The/^econdyariable concerns the type of de scri ptor cont ent^col- 
Iected. ^FbatisTwhat is the rationale for using a~3es£riptor type (e.g., 
work activities, KSAOs, competencies, work context, performance stan- 
dards), or particular mix of descriptors? For example, a given type of 
information may be essential for establishing the content validity of a 
selection procedure according to the Uniform Guidelines on Employee 
Selection Procedures (EEOC, 1978) but may be completely unnecessary 
for conducting job evaluation research. 

^fiiir d)what are the procedures for developing descriptor content 
thatwill form the basis ot the work model being crearBclv Is content 
simply selected from a library of descriptor statements after a brief walk- 
through and observation of the target jobs? Is it a preexisting and fixed 
inventory of content selected after researching the literature? Are there 
extensive interviews and focus groups held to develop a more refined set 
of descriptor content?. . . and so forth. A defining characteristic of this 
variable is the extent to which a meaningful sample of content experts is 
involv^don selecting or developing the descriptors. 

(^r^urth^hat is the detail of the descriptor conten t? Does the final 
soWirjrTconsist of simpfy a handful of broad labels' representing cate- 
gories of content with no associated definitions or are the content cate- 
gories precisely and unambiguously described using crisply defined sets 
of homogeneous item-level content descriptors that leave no room for 
misinterpretation? Furthermore, to what extent are the content cate- 
gories free from overlap, and to what degree are the item-level descrip- 
tors written at the same level of detail (i.e., narrow vs. broad) within and 
acrossxategories? 

{^Rfth^how clo sely are the r esea rch resu l ts linked to business jpals 
an5s§atfigies? In other words, to what extent has there been an effort 
to identify and link to the broader organizational goals and long-term 
strategies of the organization? For example, a platform of information 
might be collected to guide the creation of a testing program for a job 
group, describing the work activities and associated competency require- 
ments as they exist now, without any attempt to factor in the reality that 
the organization is changing its business strategy. More specifically, a 
finance company may be promoting a shift from a passive, responsive 
orientation to sales to a more proactive selling approach with an empha- 
&s^npromoting tag-ons. 

Sixty how extensively are the organization's job content expertejn - 
vglylEUn some form of content review of the resulting categories and 
subsumed item-level descriptors? That is, subsequent to the creation of 
the rationally derived taxonomy, who and how many people provide in- 
put and feedback on descriptor content meaning, practicality, and over- 
all relevance for the target research population? 
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S^venth^to what extent is there effort to provide ajranjring of de^ 
scri^or^ontentT^FOT example, is the solution an unpnOTkized set of 
narrative content describing the target job group, or is there some form 
of rational or empirical procedure used to prioritize the categories and 
items in terms of importance for job success, frequency of occurrence, 
difficuljyto learn, or some other scale? 

/Sghtoi to what extent is there an effort to assess the consistency or 
r eprodticibii^ the resuitingjsolution or of the judgments that resulted 
ifTthe creaHorTof the finaldesaiptor set? Reliability here can be in 
terms of the classification of item-level descriptors or behaviors into 
broader categories or dimensions, or in terms of the ranking of items 
and/or dimensions according to some scale (e.g., importance, frequency, 
difficulty). 

^TSJintj 5)to what extent are there clear and logical criteria for includ- 
ii^oLex^ding^ 

final descriptive set (i.e., solution or model)? In other words, from the 
universe of potentially useful descriptors for a target job, are there ra- 
tionally developed and consistendy employed criteria for retaining or 
deleting content in order to create a more rarefied final set? These cri- 
teria might be some minimal level of agreement among SMEs about the 
behaviors that relate to successful performance, a basic level of impor- 
tance or frequency, a measure of linkage to business strategy, some form 
of retranslation to determine clarity, or other factors. 

(Tbrrn), to what extent is there effort to proyide^^ 
t he~Tgsearch apprc& gh, .methods used, e^OTTinvoIvedTand results ob- 
tained? Clearly, this is not quite the same type of evaluative criterion 
as the previous nine, in that variability along this dimension does not 
necessarily affect the quality of the inferences made from the product 
of the research. However, because it is so important for establishing 
the long-term credibility of the research and because it is an area with 
relatively wide discrepancies between the different approaches, it was 
judged worthwhile to include here as an evaluative criterion. 

In summary, the 10 evaluative criteria noted above represent a work 
product designed to make explicit recommendations for practice. The 
chart presented in Tkble 1 is an effort to operationally define the range of 
potential practice for each of the 10 evaluative criteria and to elaborate 
on the definitions provided above. During the 12 months this scale was 
under development, the members of the JACMTF took special care to 
ensure the definitions and examples in each ceil adequately represented 
the purported criterion being defined. 

Taken together, these evaluative criteria may be considered a mea- 
surement yardstick for comparing different approaches for gathering 
information about a position, job, job group/family, role, collection of 



TABLE 1 
Level Of Rigor Scale 



Variable 



Low rigor 



2 

Low/Medium 
rigor 



4 

Medium/High 
rigor 



1. Method of The same method for Same two methods used Variable combination Variable combination of 

investigation collecting information every time regardless of of two methods used, two or three methods 

(eg., focus groups, the research setting and depending on some used, depending on the 

observation, interviews, intended application, effort to consider the research setting, target 

or fixed-content constraints of the population, and 

research setting, intended application. 



of setting or target 



A variable combination 
and logically selected 
mix of multiple 
methods are used to 
obtain information 
(e.g., interviews, focus 
groups, observation, 



ion, 
lication, 



I Type of Same type of Same two types of Variable combination Variable combination of 

descriptor information (e.g., information collected of two types of two or three types of 

content competencies, work every time, regardless information collected, information (e.g., 

collected activities, KSAOs, or of intended depending on the competencies, work 

performance standards) applications), intended application(s), 

collected every time, 

regardless of intended collected, depending on 

application(s). the intended applies 



Variable combination of 
multiple types of 



on 
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1 


2 


J 


4 S 

-r *- 5 , - -,- 




Low/Medium 




Medium/High 


triable Low rigor 


rigor 


Medium rigor 


rigor High rigor 


3, Procedures No effort to gather 


Information is 


Information is 


Information collected Information collected 


for information from 


gathered from 


collected from a large 


from content experts from content experts 


developing content experts; 


convenient samples 


number of content 


using a structured using a structured 


descriptor instead, the researcher 


of content experts 


experts using a semi- 


protocol and with protocol and 


content or analyst serves as 


using ad hoc or 


structured protocol, 


reference to a fairly following a logically 


sole content expert, 


unstructured procedures, Some effort is made 


well thought out developed sampling 




No qualification 


to identify individuals 


sampling plan. plan with a 




criteria (e.g., time on 


most qualified to serve 


Content experts meet comprehensive and 




the job, top performers 


as content experts. 


some qualification representative sample, 




based on appraisals) are 




criteria (e.g., time on Content experts meet 




used to identify 




job, top performers some qualification 




individuals in the 




based on appraisals), criteria (e& time on 




best position to serve 
as content experts. 




job, top performers 
based on appraisals). 



4. Detail of Handful of broad labels Broad labels with Moderately specific Fairly precise labels A number of precise 

descriptor representing categories narrative definitions labels representing representing different labels representing 

content of content, with no or a small sample of different categories of categories of content discrete categories of 

associated definitions, descriptor items serving content and a mix of that subsume fairly content that subsume 

as the operational descriptor items helping comprehensive sets of very comprehensive 

definition. to operationally define item-level descriptors and crisply defined sets 

each category. which operationally of item-level descriptors 

define each category, which operationally define 
each category and leave 
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1 


2 


3 


4 


5 






Low/Medium 




Medium/High 




Variable 


Low rigor 


rigor 


Medium rigor 


rigor 


High rigor 


5 link to 


No attempt to 


Minimal effort to 


Modest effort to 


Substantial effort to 


Significant effort to 


business 


understand business 


research the business 


research the business 


research the business 


research the business 


goals and 


context or broader 


context and review 


context and review 


context and review 


context and review 


strategies 


goals and lone-term 


strategy-related 


strategy-related 


strategy-related 


strategy-related 




strategic of the 


documents to ensure 


documents to ensure 


documents, as well as 


documents, as well as 




organization. 


results are aligned with 


results are aligned with 


meetings with HR 


meetings with top 






the broader goals and 


the broader goals and 


and/or line managers 


executives rfisnonstble 






long-term strategies of 


long-term strategies 


who are aware of the 


for setting strategies, 






the organization. 


of the organization, 


organization's plans, in 


to ensure the results are 










an effort to ensure the 


aligned with the broader 










results are aligned with 
the broader goals and 
long-term strategies of 


goals and long-term 
strategies of the 
organization. 


6, Content 


No content review. 


Brief review of 


Formal review of 


the organization. 
Formal review of 


Formal review of rationally 


review 




rationally created 


rationally created 


rationally created 


created solution by client pro- 






solution with the client 


solution by client 


solution by client 


ject leaders, technical review 






sponsor to ensure: 


project leaders to 


project leaders and a 


team, and potential end 






•Item-level descriptors 


ensure: 


technical review team 


users of the application built 






are clear. 


♦Item-level descriptors 


to ensure: 


from the solution to ensure: 








are clear. 


•Item-level descriptors 


•Item-level descriptors 








•Content categories do 


are clear. 


are clear. 








not overlap. 


•Content categories 
do not overlap. 


•Content categories do 
not overlap, 










•Content categories 
are parsimonious and 


•Content categories 
are parsimonious and 



internally consistent, internally consistent 

•Items and categories 

represent measurable 

content appropriate 

for the intended 
application. 
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Variable 



Low rigor 



2. 

Low/Medium 
rigor 



Medium rigor 



rigor 



High rigor 



7, Ranking None. The final Based on participation Based on some Some mix of 

descriptor descriptor set is an in interviews, focus procedure for weighting 

content unprioritized set of groups, etc, the the judgments of a 

narrative content researcher or analyst small group of content 

describing the target serves as expert and experts (e,g., limited 

rationally creates an distribution 

ordinally prioritized questionnaire), an 
descriptor set of 
broad labels. 

set is derived. 



Some mix of 



electronic monitoring, 
observation, diary data observation, diary data 



ding, or other recording, or other 

methods are used with methods are used with 

a fairly large sample of a comprehensive and 
content experts to 

lis 

I to collect data that is used 

create an interval to empirically create an 
inH 



8. Assess- No effort to assess Based upon discussions A convenient/casual 

mentof consistency or with a convenient/ sample of content 

reproducibility of casual sample of experts perform some 

the results, content experts, the rating task (e,g, rating content category labels 

analyst concludes that items on relative with definitions and 

there is general importance for successful perform some rating 

agreement among the job performance), Results task (e.g., rating items 

expertise regarding are expressed in terms of an relative importance 

the meaningfulness the average intercor- for successful job 

and relevance of the relation of the ratings, 
cal 



iple systematic 

of content experts are samples of content 



matching content 
category labels with 
definitions and item- 
level descriptors and 



structured rating task 
(e.jj. ( rating items on 



performance), Results 
are expressed in terms 
of percent correct matches successful job performance), 
and the average inter- Results are expressed 
correlations of the items, in terms of percent of 
correct matches and the 
average intercorrelatjons 
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Variable 



2 

Low/Medium 
rigor 



Medium rigor 



4 

Medium/High 
rigor 



None, All created 

items/categories 

retained 



criteria 



10. Docu- 
mentation 



None. 



rion is A single clear, logical TWo clear, logical 

criterion is consistently criteria are consistently criteria are consistently 

categories to determine applied to items and applied to items and applied to items and 

retention or deletion, categories to determine categories to determine categories to determine 



though the criterion is whether content is whether content is 
somewhat unclear or retained or deleted, retained or deleted, 
inconsistently applied, 

Brief handwritten notes, Summary file memo Standardized final 
referencing related user report "shell" 
document with 



whether content is 
retained or deleted. 



Detailed and customized 
written report which 
thoroughly describes the 
appropriate information procedures employed and 
and data slotted in, the composition of content 



copies of instruments used, 
and comprehensively reports 
the results. 
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titles in a value chain, and so forth. In the current initiative, the in- 
tent was to provide a comparison between competency modeling and job 
analysis. Using (a) the results of the literature review, (b) the write-ups 
from the 37 SME interviews, and (c) the numerous discussions among 
task force members concerning the hundreds of job analysis and compe- 
tency modeling studies the authors have reviewed over the years as the 
information base, 11 members of the JACMTF completed a rating task 
using the level of rigor scale applied to both competency modeling and 
job analysis. 

To elaborate, prior to conducting ratings, the literature review was 
used to uncover important source documents, which were then copied 
and distributed to task force members. The import and implications of 
these documents were then discussed and debated over numerous con- 
ference calls over a 2-year time frame. The sum of these written doc- 
uments and the associated discussions represent one source of stimulus 
material used by the task force members in making their judgments. In 
addition, the results of the literature review were used to identify poten- 
tial content experts to include in the SME interviews. 

The SME interviews constituted a second leg of the information base 
used by the JACMTF raters. The juxtaposition of responses from the 
six perspectives represented in the interview protocol were consolidated 
into a single document so task force members could meaningfully com- 
pare and contrast the different perspectives. This presentation of the 
SME interview results proved to be particularly valuable and was made 
available to task force members prior to making their ratings. A slightly 
modified version of this document, which disguises the identity of who 
said what (SMEs were guaranteed anonymity to ensure that their frank 
responses did not come back to haunt them in their current organiza- 
tions or in efforts to consult with clients), is available from the task force 
chair. 

Finally, the members of the JACMTF themselves represented a 
wealth of experience in both competency modeling and job analysis. The 
more personal experiences and projects were shared openly among the 
members of the task force, and the observations of what we were run- 
ning into in the field, what had (or had not) worked in our own research, 
and so forth, were frequently discussed over the course of 2 years of 
conference calls. These discussions served as a third source of shared 
information and context from which raters provided judgments. 

Based on everything we had learned, instructions to task force mem- 
bers were to make their j udgments with reference to the typical or modal 
level of practice on each of the 10 criteria for both approaches. Means, 
standard deviations, and correlations between the 10 criterion variables 
appear in Table 2. Figure 1 provides a graphic representation by juxta- 
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TABLE2 

Evaluative tiittw: Descriptht Statistics and CmMou 



Competency modeling 



Variables 


M 


SD 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


Competency modeling 
1. Method of investigation 


2,00 


,89 






















2. Type of descriptor 


1.73 


1.10 


63 




















3, Procedures for developing 


2,64 


1.21 


19 


22 


















4. Detail of descriptor 


2.09 


.54 


21 


32 


82 
















5, Link to business 


4.45 


52 


21 


-17 


29 


19 












| 


6, Content review 


2.90 


.99 


00 


-21 


60 


61 


52 












7, Ranking descriptor 


127 


.79 


00 


26 


64 


64 


-33 


31 










g, Assessment of reliability 


1.73 


.90 


49 


75 


18 


06 


-13 


-39 


26 








9. Retention criteria 


240 


.97 


12 


57 


57 


32 


-13 


05 


53 


75 




s 


10. Documentation 


3.36 


1.36 


49 


57 


64 


50 


31 


27 


55 


58 


63 




lob analysis 


























1. Method of investigation 


3,73 


.90 


00 




















2. Type of descriptor 


3.82 


.87 


-13 


-03 
















1 


3, Procedures for developing 


4.09 


.54 


-41 


-42 


21 
















4. Detail of descriptor 


4.00 


.63 


-18 


-25 


13 


00 














5. Link to business 


227 


.65 


-36 


-30 


27 


50 


-11 












6, Contentrcview 


370 


.82 


-71 


-53 


23 


07 


31 


23 










7. Ranking descriptor 


3.55 


.82 


-55 


-48 


-59 


-58 


-40 


-45 


-10 








8. Assessment of reliability 


3.55 


.93 


00 


09 


-61 


-51 


-56 


-63 


-09 


31 






9. Retention criteria 


4.10 


.74 


-32 


-08 


-32 


-29 


-12 


-29 


-05 


37 


25 




10, Documentation 


4.64 


.50 


22 


36 


42 


50 


31 


13 


28 


20 


13 


36 
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lob analysis p 

tables M SD 1 2 3 4 5 6 7 8 9 < 
_ — „ _ , w 

Job analysis ^ 



1. Method of investigation 


3.73 


.90 
















2. Type of descriptor 


3.82 


,87 


82 














3, Procedures for devetopini 


5 4.09 


.54 


67 


46 












4. Detail of descriptor 


4.00 


.63 


70 


54 


88 










5. Link to business 


127 


.65 


-03 


-08 


-09 


-25 








6. Content renew 


3,70 


.82 


31 


57 


51 


31 


18 






7. Ranking descriptor 


3,55 


.82 


22 


29 


10 


00 


26 


29 




8. Assessment of reliability 


3.55 


.93 


19 


26 


-11 


00 


-11 


-17 


75 


9, Retention criteria 


4.10 


.74 


43 


71 


-07 


-03 


16 


42 


79 


10. Documentation 


4.64 


.50 


20 


29 


-23 


00 


03 


21 


-20 
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PERSONNEL PSYCHOLOGY 



VARIABLE 


i 


2 


3 


4 


S 


Low Rigor 


Low/Mediuni 
Ripor 


Medium Rigor 


Medium/High 
Rigor 


High Rigor 


I no-1 75 


1 .75^2.50 


25 1-150 


3 .51-4 25 


4.26-5.00 


1. MeihodOt 
Investigation 




LWI4» A 








2. Type Of 
Descriptor 
Content 
Collected 








Q >.S2/g,S7 

1 

1 
1 




3. Procedures 
For 

Developing 
Descripior 
Cement 








-p- 
1 
r 

P 

i 

i 




4. Detail Of 
Descriptor 
Content 








— | ' 

j 




5. Link To 
Business 
Goals And 
Strategies 












6. Content 
Review 












7. Ranking 
Descriptor 
Content 












S. Assessing or 
Reliability 


t,T3/».» 










9. Item/Category 
Retention 
Criteria 








4.W0.T4 *^ 




10. Documentation 













♦Means/standard deviations are reproduced next to plotted points in the figure. 

~ 'Typical" compentency modeling project 

"Typicar multimethod jot? analysis project 



Figure 1: Comparison of *TjpicaP' Competency Modeling Rigor Profile Versus 
"Topical* Job Analysis Approach* 
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posing the snakeplot of means for both competency modeling and job 
analysis. 

Of course, the snakeplot profiles presented in Figure 1 in no way rep- 
resent "the truth." A single point on this scale cannot capture the range 
of activities that occur along the practice continuum, in both the compe- 
tency modeling and job analysis camps, associated with each evaluative 
criterion. In short, neither job analysis nor competency modeling are 
singular approaches to studying work. Furthermore, competency mod- 
eling practice in particular is a quickly evolving approach. What might 
be considered typical practice today may well be different 5 years from 
now. Nevertheless, with these cautions against over -interpretation in 
mind, the JACMTF does believe the data in Tkble 2 and the juxtaposed 
snakeplots in Figure 1 reasonably illustrate the preponderance of prac- 
tice in these two approaches to capturing information about jobs and the 
associated person requirements at this point in time* 

As Figure 1 indicates, job analysis approaches are, in general, likely 
to be more rigorous in managing and controlling the variables that can 
impact the quality of the resulting information than competency model- 
ing approaches. The most pronounced areas of distinction include vari- 
able 2, the type of descriptor content collected, and variable 8, assess- 
ment of reliability. In the first case, competency modeling projects are 
much more likely to use the same type of input content (Le., competen- 
cies) every time, regardless of the research setting and intended appli- 
cation^) to be constructed, whereas job analysis approaches are more 
likely to use variable combinations of two or three types of input con- 
tent (e.g., tasks and/or KSAs or competencies and/or performance stan- 
dards) depending on the unique demands of the research context and 
the intended application(s). In the second case, the majority of compe- 
tency modeling efforts are unlikely to include any effort to evaluate the 
consistency or reproducibility of the results. 

The one area where the typical competency modeling approach is 
likely to demonstrate more rigor than its job analytic counterpart is with 
reference to variable 5, establishing a link to business goals and strate- 
gies. Competency approaches typically include a fairly substantial effort 
to understand an organization's business context and competitive strat- 
egy and to establish some direct line-of-sight between individual compe- 
tency requirements and the broader goals of the organization* On the 
other hand, examples of job analysis methods endeavoring to make this 
connection are few and far between. 

It should be noted that the area of most disagreement among the 
JACMTF raters involved variable 10, which concerns the level of doc- 
umentation in competency modeling (SD - 1.36). The ratings ranged 
from 1 to 5 on the 5-point scale and reflect the two distinct points of 
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view among the task force members. On one side were those who felt 
our input data indicated that the typical competency approach yielded 
minimal, if any, formal documentation of methods and results. Con- 
versely, there was a contingent from the task force that viewed com- 
petency approaches as often producing the kind of comprehensive and 
thorough reports that clearly characterize job analysis methods (X - 
4,64; SD = .50). Although only conjecture, it may well be that this lack of 
agreement reflects the degree of flux in the practice of competency mod- 
eling. Several years ago, the typical back-end of a competency modeling 
project probably did result in minimal documentation, although today 
the level of reporting is less distinguishable from job analysis approaches. 
In part, this perceived trend may be a result of more I-O psychologists 
conducting work in the domain of competency modeling; documenting 
the development of their models in a manner similar to which they have 
grown accustomed when doing job analysis research. 

The JACMTF raters demonstrated a similar lack of agreement in the 
ratings of variable 3, the procedures for developing descriptor content. 
Again, the lack of agreement probably reflects, at least in part, the evolu- 
tion of practice in competency modeling. Several years ago it was fairly 
common to see competency models developed solely based on a "mod- 
eling expert" rummaging through the organization for a day or two and 
then simply selecting competency dimensions from an existing compe- 
tency library. Our sense is that, more and more, researchers working 
under the rubric of competency modeling are likely to collect informa- 
tion from an organization's own content experts, follow some form of 
logical sampling plan, and use some type of structured protocol. 

Why Are Competencies So Appealing To Consumers? 

The above comparison of competency modeling and job analysis, 
although informative, tells only part of the story. There are other points 
of comparison that could be made between competency modeling and 
job analysis. Examining some of these "other" variables, in addition to 
further defining the competency modeling approach, may provide some 
insight into the market appeal of competency methods. 

The first of these additional, nonevaluative variables concerns the 
extent to which there is a focus on core competencies. For example, 
to what degree do the different approaches (i.e., competency modeling 
versus job analysis) endeavor to identify and describe what is similar 
across jobs, business segments, functional areas, job levels, and so forth? 
Second, to what extent is there an effort to identify and document the 
functional knowledge or technical skills associated with a job? Third, 
is there an attempt to identify those areas of content that are related 
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to long-term organizational fit versus short-term job match? Fourth, to 
what extent is there an attempt to include personal value and personality 
orientations in the mix of descriptor content defining a job, work role, 
job group, or whatever? Fifth, to what extent are the category labels and 
subsumed descriptor content face valid and written in the language of 
the client organization? Sixth, to what extent is the approach used to 
drive training and development applications? Seventh, to what extent is 
the approach used to drive selection, performance appraisal, and other 
human resource decision making applications? 

As with the evaluative criteria discussed in the previous section, the 
above seven variables may be used to compare and contrast the "typi- 
cal" competency modeling and job analysis approaches. Similar to the 
procedures described previously, 11 members of the JACMTF rated 
both competency modeling and job analysis with reference to the seven 
"other" variables described above. However, in this case, the seven vari- 
ables were rated using a scale anchored by "expressions of amount" 
(e.g., 5 - a significant extent, 4 - quite a bit, 3 « some, 2 ™ a limited 
extent , 1 = hardly any or none). Thus, the rating task involved making a 
judgment about the extent to which each of these seven additional vari- 
ables characterized the typical competency and the typical job analysis 
approaches; again, the (a) literature review, (b) SME interviews, and (c) 
collective experiences of task force members were used as the basis for 
the judgments. Means, standard deviations, and correlations between 
the seven additional variables appear in Table 3. Figure 2 graphically 
presents a portion of the results by juxtaposing the snakeplot of means 
for both competency modeling and job analysis. 

As Figure 2 illustrates, there are a number of pronounced differ- 
ences between competency modeling and job analysis, though none may 
be more telling than the first. Competency modeling approaches typi- 
cally provide descriptions of the individual-level competencies that are 
core, or common, for an occupational group, entire level of jobs (e.g., 
executive, management, supervisory, hourly), or for the organization as 
a whole. The focus is on broad applicability and leveraging what is in 
common or universal. Even when the modeling effort targets a narrowly 
defined job group (e.g., sales managers), the resulting descriptions are 
typically at a fairly high level and general in nature. The focus for job 
analysis, on the other hand, is typically description at a level that empha- 
sizes what is distinct or different across jobs, occupational groups, levels 
in the organization, and so forth. Although job analysis can at times take 
a broad focus (e.g., when conducting job family research), the descrip- 
tor items serving as a basis for the grouping typically represent a level of 
granularity that is far more detailed than is achieved by most competency 
modeling efforts. 
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Job an 


ialvsis 


1 


2 


3 


4 


5 


6 


7 1 


2 


3 


4 5 6 


Competency modeling 


























1. Core competencies 


4.00 


.89 






















2. Technical skills 


2.00 


1,00 


-34 




















3. Fit versus job match 


3,82 


1,08 


31 


-56 


















4. Value & personality orientation 


3.91 


1.14 


49 


-44 


31 
















5. Face validity of content 


4.64 


.50 


00 


40 


05 


-06 














6. Ttaining & dev. applications 


4.55 


SI 


43 


-19 


02 


43 


45 












7. Selection & decision applications 


m 


1.04 


11 


29 


-16 


01 


-31 


-10 










Job analysis 


























1. Core competencies 


2.45 


1.13 


-40 




















2. Tfccbnical skills 


4.55 


.52 


43 


00 










-63 








3, Fit versus job match 


2.55 


1.13 


79 


-27 


17 








-45 


29 






4. Value & personality orientation 


2.45 


1.44 


23 


00 


19 


-34 






-02 


04 


20 




5. Face validity of content 


3.18 


1.17 


10 


-09 


-05 


-44 


-22 




-07 


-34 


37 


60 


6. Ttaining & dev. applications 


3.55 


1.04 


11 


19 


01 


-12 


42 


13 


-40 


32 


40 


49 49 


7. Selection & decision applications 


4,36 


1.12 


-50 


09 


-44 


11 


-10 
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VARIABLE 


1 


2 


3 


4 
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Figure 2: Comparison of "Typical" Compentency Modeling and "Topical" job 
Analysis Approaches on Seven "Other" Variables 
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Clearly, the emphasis on core competencies has been an important 
selling point for competency modeling. On the downside, models built 
using only general competencies tend to look alike, despite being devel- 
oped for different jobs, functions, or organizations. The flip side of the 
coin is whether an approach tends to include technical skills in the re- 
sulting description of a job, job group, role, or whatever. As variable 2 
(focus on technical skills) indicates, this is often part of the focus for job 
analysis approaches, but less so for competency modeling. 

Further, competency modeling approaches are more likely to em- 
phasize long-term organizational fit versus shorter-term job match con- 
tent in the resulting descriptions. This emphasis is likely a natural con- 
sequence of the effort to incorporate an organization's vision and core 
values in part of the modeling process as described under the fifth eval- 
uative criterion in the previous section. Similarly, value and personality 
orientations are more likely to be built into the models that are pro- 
duced. 

The face validity of the content, variable 5, is another differentiator 
between the two approaches. More so than job analysis approaches, the 
typical competency modeling project is likely to include various review 
sessions and focus groups to ensure the descriptive content captures the 
language and spirit that is important to the client organization. As a 
result, the consumers "see themselves" in the resulting descriptions, an 
outcome that many job analysis approaches don't come close to emulat- 
ing. This is not a small distinction when looking for reasons to explain 
the surge of popularity in competency modeling. 

Finally, though there is some disagreement among the JACMTF 
raters as indicated by the magnitude of the standard deviations, it ap- 
pears as though competency modeling approaches are slightly more 
likely to serve as a platform for training and development applications, 
and for job analysis approaches to be more likely to serve as a basis for 
creating selection, performance appraisal, and other human resource 
decision making applications. More than anything, this distinction is 
probably driven by the recognition that the level of rigor and documen- 
tation found in many competency modeling efforts would have difficulty 
withstanding the close scrutiny of a legal challenge. 

What Js The Future Of Competency Modeling? 

As part of the SME interview process, individuals were asked to de- 
scribe the future of competency modeling. They provided a range of an- 
swers with no clear theme emerging. Some see an increasing number of 
organizations using the approach. This excites one group of SMEs who 
see possibilities for improved practice, but it concerns another group of 
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SMEs who worry about a diluted emphasis on methodology and rigor in 
model development. Representatives from both camps believe that the 
work will be more technologically driven with improved software appli- 
cations (e.g., expert systems that help develop models). Though by no 
means a majority, a few participants see absolutely no value in compe- 
tency modeling and expect it to die quickly. 

One thing we can say for certain is, as with other management trends 
that have influenced the human resources field in recent times (e.g,, 
work teams, total quality management, reengineering), the field of I-O 
psychology has not led the competency modeling movement, despite the 
fact that defining the key attributes needed for organizational success is 
a "core competency" of I-O psychology. Rather than helping to steer 
the course for practice, applied psychologists have been merely riding 
the wake of yet another trend. 

Where do we go from here? At the minimum, our hope is that this 
article helps explain and define a significant trend — competency mod- 
eling—that many in the I-O field have had trouble conceptualizing and 
understanding. In an effort to facilitate understanding, we have chosen 
to compare competency modeling to something somewhat similar in na- 
ture that most I-O psychologists understand quite well: job analysis. 

However, our intent is to not simply stop at this point Although the 
stated charter of the task force did not include developing detailed pre- 
scriptive guidelines for practice, we do believe the level of rigor scale 
that was developed as a basis of comparison for the competency and job 
analysis approaches could be extended and used as a structure for estab- 
lishing guidelines. The appropriate interpretation of the results from a 
front-end information gathering methodology, such as competency mod- 
eling or job analysis, depends on a number of assumptions. The extent 
to which these assumptions are considered and controlled will influence 
the degree of trust one has in the results. Of course, the minimum level 
of rigor required for each of the 10 evaluative variables listed in Tkble 1 
has not been established as part of the work of this task force. This could 
be a logical next step in an effort to make more concrete the standards 
for practice that do exist in documents like the Uniform Guidelines on 
Employee Selection Procedures (EEOC, 1978), Principles for the Vali- 
dation and Use of Selection Procedures (SIOP, 1987), and the Standards 
for Educational and Psychological Tfesting (AERA, APA, & CME, 1999). 

However, it should be pointed out that the minimum levels of rigor 
required in each of the 10 evaluative criteria may be different in different 
situations. In other words, there is an array of contextual and practical 
variables that can impact decisions about the level of rigor that is most 
appropriate in a given situation. For example, "impact" variables like 
the: 
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• Purpose or target application(s) of the research (e.g., job evaluation, 
selection, job classification); 

• Extent to which the target application is expected to be free from 
legal or union review; 

• Number of specific job titles (and incumbents) included in the scope 
of the proposed application; 

• Extent to which there are planned changes in the organization that 
will likely affect the work process or performance requirements of 
targeted jobs; and, 

• Extent to which the results of work in the targeted jobs are observable 
and/or dynamic; 

can greatly influence the level of rigor required in evaluative criterion 
variables like the: 

• Appropriate method of investigation; 

• Type of descriptor content collected; 

• Procedures for developing descriptor content; 

• Detail of descriptor content; and the, 

• Linkage to business goals and strategies. 

So, rather than overly general "minimum level of rigor" recommenda- 
tions, what is required at this point is research that investigates the pro- 
files of influence different impact variables have (or should have) on the 
level of rigor requirements for different evaluative criteria. Schippmann 
(1999) provides a list of impact variables and a description of initial re- 
search resulting in situation assessment profiles for different HR appli- 
cations. 

In addition, our hope is that this article will serve as a wake -up call 
for researchers conducting job analysis work (both applied and basic). 
In an era of unprecedented change in business and the way work is 
structured, when organizational leaders eagerly seek useful information 
they can use to drive decision making, it is fascinating that a powerful 
tool like job analysis is widely viewed as a mundane technical activity 
(Prien, 1977) that is merely part of the critical path toward developing 
some other really useful application. Perhaps it is time we updated our 
thinking of job analysis and regarded the approach as more an ongoing 
OD intervention than a loosely connected series of tactical projects that 
happen to occur under the roof of a particular organization. From this 
perspective, the practice of job analysis is in a position to learn something 
from the upstart called competency modeling. 

Specifically, the field of job analysis might benefit from paying closer 
attention to the strategic and future-oriented needs of organizations. It 
is reasonable to conclude that decisions about the strategy and direction 
of an organization will have downstream implications for work roles and 
worker requirements. To paraphrase Scheider (1976), work roles, job re- 
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quirements, and organizations do not exist independently. Yet there is 
virtually no research and very little description of practice. How might a 
job analytic researcher go about documenting the ultimate objectives of 
an organization and translating this information into work requirements 
for a job or job group? Does the concept of a Strategic Relevance Ra- 
tio, somewhat along the lines of Lawshe's (1975) Content Validity Ratio, 
make sense as a way to measure the distal linkage between job analytic 
results and the higher order goals of an organization? Opportunities for 
research and creative practice abound and job analysis practitioners ap- 
pear to be overlooking a terrific opportunity to consult with, and deliver 
value to, an organization's senior leadership team* 

Next, as noted earlier in the paper, many job analysis approaches are 
focused on describing the content of a job or identifying what is different 
between jobs. What seems to be frequently overlooked in job analysis 
research is to identify what is common across jobs, job groups, occu- 
pational groups, business segments, and so forth, in an effort to build 
platforms of information that can be used to support a broad range of 
applications in an HR system. In other words, what activities or worker 
characteristics are core or important organization-wide? Or, for a par- 
ticular organization, what descriptive content of jobs or workers might 
be business unit specific (e.g., are some general skills or orientations im- 
portant in international business segments but not domestic ones?). Is 
it possible to map out and chart changes in the common requirements 
associated with jobs across levels (e.g., which activities or worker charac- 
teristics are components of all executive jobs vs. midlevel management 
jobs vs. entry-level supervisor jobs)? Further, how might these broad 
or general classes of descriptive content be aligned with the very de- 
tailed, technical, functional-specific activities, skills, and knowledge de- 
scriptions that are required to drive some HR applications? 

By posing these questions we do not mean to imply that the folks in 
the competency modeling camp have got everything figured out. This 
clearly is not the case. However, in the information-driven organization 
of today, where dramatic developments in HR software and enterprise- 
wide resource planning systems (ERP) are reshaping the role of HR 
(Schippmann, 1999), they appear further down the path than those in the 
job analysis camp. These ERP systems (from companies like PeopleSoft, 
SAP, Oracle, Baan, and Lawson, to name just a few) require underlying 
definitions and architectures of work and worker requirements. In this 
arena, competency advocates have been quick to try and build models 
that lay the foundation for integrating multiple HR applications. If the 
job analysis practitioners are not careful, they may find themselves left 
out of the picture in what is shaping up to be a huge development in HR 
management. Rather than redirecting the focus of job analysis prac- 
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titioners, the suggestion here is an expansion of focus, to place equal 
emphasis on documenting, analyzing, displaying, and in general think- 
ing about what is core or common across jobs, job levels, functions, and 
business groups in an effort to support integrated systems of HR appli- 
cations. In short, the more rigorous job analysis techniques may also be 
used to define core competencies. 

The inclusion of variables such as personality and value orientations 
into the mix of potentially useful descriptive content is another oppor- 
tunity for positive change for job analysis; and one that has been noted 
elsewhere (Guion, 1998; Raymark, Schmit, & Guion, 1997). The goal 
here would be to identify and measure those characteristics that tap into 
an individual's willingness to perform certain activities or to "fit in" with 
the work culture of a particular organization. These are organization- 
ally hot topics today, and many job analysis approaches have been slow 
to build these perspectives into the descriptive content that characterize 
the outcome of job analytic research. 

The flip side of the above discussion is equally useful, and that is to 
elaborate on how competency model development can benefit from job 
analysis methodology, lb begin, a common pitfall of many competency 
modeling efforts appears to be an attempt to overgeneralize from the 
core competency concept Not all of the competency items or behaviors 
that operationally define a competency dimension are equally useful 
for all jobs, job levels, business segments, or regions throughout the 
organization. Increased attention on procedures for prioritizing some 
content, so the resulting models reflect the unique needs of different job 
groups and areas of the organization, will likely produce results that are 
more meaningful. In this regard, job analysis methodology offers a wide 
range of qualitative and more empirical procedures for weighting the 
content of a descriptor set. 

Similarly in contrast to competency modeling, many job analysis ef- 
forts include at least some effort to evaluate the consistency or repro- 
duceability of the results. If competency modeling initiatives included 
some evaluation of the resulting categories in terms of the percentage 
of behaviors correctly matched to competency category definitions by 
users of the final system or target application, or examined the average 
intercorrelations of items within a category, or investigated the agree- 
ment between SME raters (which would be possible if there were some 
method for prioritizing content — see above), it would likely help open 
doors for using modeling results to support HR decision-making appli- 
cations such as selection and performance appraisal. By the same token, 
it would probably also help if there were some increased attention on the 
part of competency modelers to document their procedures for creating 
(or selecting) descriptor content (e.g., rules for establishing the qualifi- 
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cations of content experts, logic behind sampling plans). Although we 
are not aware of any legal challenges sustained by organizations using 
competency models to support their HR applications, it can be expected 
that, as such challenges start to reach the courts, the need for better doc- 
umentation will become more obvious. 

Next, although not wanting to recreate what sometimes seems to be 
a myopic focus on task statements and technical skills found in job anal- 
ysis, the competency movement might benefit from efforts to integrate 
technical or functional competencies into the resulting models. In short, 
focusing only on broad, general competencies leaves a large portion of 
what is related to an individual's success in a job unaccounted for. Per- 
haps a modularized approach is possible, where it is possible to mix and 
match subsets of general competencies and technical competencies to 
create more comprehensive descriptions of a target population (which 
may be defined as broadly or narrowly as need be to serve a particu- 
lar purpose). For example, perhaps only those subsets of the general 
competency domain that are "core" should be used to communicate the 
agreed-upon guiding values and principles of the organization. On the 
other hand, perhaps a mix of the most important core competencies and 
technical competencies might be used to develop selection specifications 
for a particular job or job group. 

In addition, in our view, the resulting models or descriptive tax- 
onomies will be most powerful, and have the greatest utility, when the 
level of detail provided by the approach matches that required for use. 
However, there are no specific rules to guide practice in this area, for ei- 
ther competency modeling or job analysis, and there should be. In a re- 
lated vein, the operational models used in organizations need to have the 
flexibility to allow users to drill down to a level of detail required to sup- 
port certain applications, and to spiral up to a broader or more generic 
set of descriptors to drive other applications where the additional detail 
is unnecessary or a distraction. 

Finally, it is the opinion of the members of the task force that no 
single type of descriptor content (e.g., competencies, KSAOs, work ac- 
tivities, performance standards) is best suited for all purposes. If the 
plan is to conduct job evaluation research, then a mix of detailed work 
activity descriptors and moderately detailed KSAO or competency de- 
scriptors may be required. On the other hand, if the intent is to create 
a behaviorally based interview guide for selection purposes, then per- 
haps a mix of moderately detailed competency descriptors and an asso- 
ciated set of performance standards would be most useful. Similarly, the 
configuration of content would likely change depending on whether one 
was creating a 360° developmental feedback tool or engaging in content- 
oriented test construction, and so on. Up until about 1970, the different 
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job analytic approaches focused primarily on a single type of information 
(Prien & Ronan, 1971). However, over the past 30 years, there has been 
growing recognition of this issue in the job analysis camp, as evidenced 
by the number of muitidomain approaches available (e.g., Drauden & 
Peterson, 1974; Lopez, Kesselman, & Lopez, 1981; Prien, Goldstein, & 
Macey, 1987). Conversely, there is little evidence of a similar level of 
recognition on the part of those working in the competency modeling 
arena. 

In conclusion, there is an old Chinese curse which, when translated, 
essentially says ''May you live in interesting times. " Well, the pace of 
change in business, coupled with the rapid advance of enterprise-wide 
computing technology and information systems, has made things pretty 
interesting in the HR field. One of the areas where practice has become 
really interesting cuts right to the heart of the field and involves how we 
go about identifying and describing the important work-related aspects 
of a job and the associated worker requirements. Although none of the 
SMEs in the interview process made this point, what the future might 
hold is a blurring of borders as the competency modeling and job analysis 
approaches evolve over time. Thus, the next generation of approaches 
in each case may result in a blending of best practices such that there are 
more similarities than the differences that exist today. 
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APPENDIX 
Interview Protocol 

1. How do you define a "competency"? (How is it different from the 
concept of KSAOs?) 

2. What types of human resource applications can competencies and 
competency modeling support? (Where have you found competen- 
cies to be most successfully applied?) 
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3. What types of human resource applications should competencies and 
competency modeling not be used to support? (Why not?) 

4. How is competency modeling different from job analysis? (What are 
the basic steps involved in developing a competency model?) 

5. Should competency models be validated? (If " no," why not? If "yes, " 
when and how?) 

6. Why competencies? What is the attraction/value beyond other ways 
individual differences might be described and characterized? 

7. Where do you see the future of competency modeling headed? 

8. Would you provide us with some background information about 
yourself so we may create a general profile of the people surveyed? 
(What is your job title? Number of years directly involved in the 
study or application of competencies? Approximate number of ap- 
plied competency projects completed?) 

9. Do you have some written material, brochures, presentation content, 
summaries or write-ups of project work, and so forth, related to this 
topic that you would be willing to share with the task force that could 
illustrate and elaborate on some of your points? 



